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On  February  18,  2009,  ITEIA  sponsored  a  workshop  on  “Future  Defense  Spending  and  the 
Implications  for  Test  Evaluation,  ”  to  focus  on  the  impact  of  the  economic  downturn  and  the 
change  of  administration  on  Defense  system  acquisitions  and  test  and  evaluation  (Tl^E).  The 
final  panel  of  the  day  featured  the  Department  of  Defense’s  senior  T^^E  leadership,  past  and 
present  (Phil  Coyle,  Jack  Krings,  Tom  Christie,  and  Chuck  McQueary).  The  four  Directors  of 
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On  February  18,  2009,  ITEA  spon¬ 
sored  a  workshop  on  “Future  De¬ 
fense  Spending  and  the  Implications 
for  Test  &  Evaluation”  to  focus  on 
the  impact  of  the  economic  down¬ 
turn  and  the  change  of  administration  on  Defense 
system  acquisitions  and  test  and  evaluation  (T&E). 

The  final  panel  of  the  day  featured  the  Department 
of  Defense’s  (DoD’s)  senior  test  and  T&E  leadership, 
past  and  present  (Figure  1 ): 

•  The  Honorable  John  E.  (Jack)  Krings,  Former 
Director,  Operational  Test  and  Evaluation; 

•  The  Honorable  Philip  E.  Coyle,  III,  Former 
Director,  Operational  Test  and  Evaluation; 

•  The  Honorable  Thomas  P.  Christie,  Former 
Director,  Operational  Test  and  Evaluation;  and 
•  The  Honorable  Charles  E.  McQueary,  Current 
Director,  Operational  Test  and  Evaluation. 

As  most  readers  know,  the  Director,  Operational 
Test  and  Evaluation  (DOT&E),  is  appointed  from 
civilian  life  by  the  President,  by  and  with  the  advice 
and  consent  of  the  Senate — as  aU  the  panelists  were  for 
their  tenure.  By  law,  the  Director  is  appointed  without 
regard  to  political  affiliation  and  solely  on  the  basis  of 
fitness  to  perform  the  duties  of  the  office  of  Director. 
The  Director  is  the  principal  adviser  to  the  Secretary  of 
Defense  and  the  Under  Secretary  of  Defense  for 


Acquisition,  Technology,  and  Logistics  on  operational 
T&E  in  the  DoD,  and  the  principal  operational  T&E 
official  within  the  senior  management  of  the  DoD. 

Defined  in  Title  10,  US  code,  section  139,  the 
Director’s  duties  are  to; 

•  set  policies  and  procedures  for  the  conduct  of 
operational  T&E  in  the  DoD;  provide  guidance 
to  and  consult  with  the  Secretary  of  Defense  and 
the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  and  the  Secretaries  of 
the  military  departments; 

•  monitor  and  review  aU  operational  T&E; 

•  coordinate  operational  testing  conducted  jointly 
by  more  than  one  military  department  or  defense 
agency; 

•  review  and  make  recommendations  to  the  Secretary 
of  Defense  on  all  budgetary  and  financial  matters 
relating  to  operational  T&E,  including  operational 
test  facilities  and  equipment;  and 

•  monitor  and  review  the  live  fire  testing  activities. 

The  Director  may  communicate  views  on  matters 
within  his/her  responsibility  directly  to  the  Secretary  of 
Defense  and  the  Deputy  Secretary  of  Defense — 
without  obtaining  the  approval  or  concurrence  of  any 
other  official  within  the  DoD. 

Clearly,  individuals  qualified  to  be  Director  are 
accomplished  persons  with  substantial  credentials  and 
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Figure  1.  The  Directors  of  Operational  Test  and  Evatuation  past  and  present  are  (left  to  right)  Phii  Coyle,  Jack  Krings,  Tom  Christie, 
and  Chuck  McOueary. 


capabilities.  Each  former  Director  brought  to  the  job  a 
unique  perspective  and  experienced  a  different  envi¬ 
ronment.  Jack  Krings  was  an  experimental  test  pilot 
and  the  first  DOT&E.  Phil  Coyle  was  a  Director  of 
the  Lawrence  Livermore  National  Laboratory,  served 
the  longest  and  under  the  most  Secretaries  of  Defense. 
Tom  Christie  was  a  defense  analyst  and  had  served 
within  the  government  the  longest  before  assuming  the 
DOT6cE  job.  Chuck  McQueary  in  the  private  sector 
was  president  of  an  advanced  technology  company;  just 
prior  to  becoming  DOT8cE  he  was  Under  Secretary 
for  Science  and  Technology  in  the  Department  of 
Homeland  Security.  The  panel  provided  us  a  rare 
opportunity  to  glimpse  a  bit  of  the  history  of  T&E  and 
also  to  gain  insight  into  what  the  future  might  hold. 
This  panel  was,  indeed,  a  special  event. 

A  panel  moderator  on  February  18  prompted  these 
Directors  with  a  series  of  questions.  Their  answers  are 
reported  here.  For  this  article,  I  do  not  include 
traceability  of  response  to  individual.  It  is  fair  to  say,  in 
most  cases,  all  the  Directors  were  in  agreement  with  each 
other’s  responses.  But  you  will  note  a  few  differences. 

1.  The  Department  faces  hard  choices  in 
acquisition.  Some  have  said  DOT&E 
results  are  erratic  and  ignored.  Are 
DOT&E  reports  given  the  right  weight  in 
acquisition  decision-making? 

There  was  considerable  agreement  that  DOT&E 
reports  are  not  as  influential  as  they  might  be. 


Evidence  offered  for  this  position  included:  (1)  many 
programs  are  in  trouble;  (2)  DOT&E  reports  such  as 
the  Annual  Report  repeat  the  same  recommendations 
year  after  year;  (3)  some  programs  use  great  inventive¬ 
ness  to  avoid  testing;  and  (4)  so  many  programs  go  to 
full  rate  production  despite  failure  to  be  declared 
effective  and  suitable. 

On  the  other  hand,  there  is  evidence  that  the 
situation  may  change.  The  new  administration  appears 
to  have  a  “fly  before  buy”  approach,  and  the  White 
House  website  includes  a  statement  on  missile  defense 
that  supports  the  approach:  “The  Obama-Biden 
Administration  will  support  missile  defense,  but  ensure 
that  it  is  developed  in  a  way  that  is  pragmatic  and  cost- 
effective;  and,  most  importantly,  does  not  divert 
resources  from  other  national  security  priorities  until 
we  are  positive  the  technology  will  protect  the 
American  public.”  The  phrase  “we  are  positive  the 
technology  will  protect”  is  a  high  standard,  consistent 
with  fly  before  buy,  and  one  that  could  require 
considerable  testing  evidence.  Another  positive  note 
is  that  the  Department  has  instituted  new  policy  that 
emphasizes  testing  and  particularly  testing  to  improve 
reliability. 

A  final  point  that  was  made  several  times  by  all  the 
panel  members:  reports  are  the  formal  means  of 
communication,  but  often  are  not  the  most  effective. 
The  most  effective  means  are  often  informal,  in  smaller 
meetings  or  in  discussions  with  decision-makers. 
Different  panel  members  had  different  relations  with 
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the  Secretary,  the  Deputy  Secretary,  or  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  hut  each  found  a  reliable  and  consistent 
audience  with  one  of  them  who  was  willing  to  listen. 
The  Director  must  bring  a  track  record  of  candor  and 
expertise  and  achieve  senior  level  relationships  needed 
to  influence  properly  the  acquisition  decisions.  This 
situation  brought  up  the  following  question. 

2.  Should  DOT&E  have  a  more  formal 
role  in  the  choices  made  to  implement 
the  slowdown  coming  in  defense  dollars? 

None  of  the  panelists  felt  that  a  formal  role  was  a 
clear  way  ahead.  DOTScE  can  offer  information  to 
those  who  need  to  make  the  decisions,  and  there  is 
hope  that  those  decisions  would  include  more 
information  on  performance  from  developmental 
testing  and  operational  testing  rather  than  just  cost 
and  schedule.  But  if  its  role  were  more  formal,  the 
value  DOT&E  can  bring  by  early  involvement  might 
be  countered  by  increased  difficulty  getting  into  the 
program. 

The  importance  of  early  involvement  was  empha¬ 
sized  by  each  panelist  as  a  way  of  realizing  several 
important  goals,  such  as  to  develop  the  formal  policy  of 
integrated  testing,  to  hopefully  generate  greater  use  of 
performance  information  in  early  decisions,  to  facilitate 
greater  involvement  of  DOT&E  early  in  a  program 
when  they  often  get  into  trouble,  and  to  promote 
greater  understanding  of  whether  the  system  is  ready 
for  its  initial  operational  T&E. 

3.  Will  budget  cuts  be  directed  at  T&E? 

The  panel  members  all  believed  that  budget  pressure 
is  likely  to  be  big  and  real.  Their  views  reflected  various 
aspects  of  the  situation.  The  actual  experience  of  the 
last  decade  and  a  half  is  indicative.  When  budgets  went 
down,  testing  was  cut;  when  budgets  went  up,  testing 
was  cut.  In  addition,  workforce  personnel  have  been 
decimated  to  the  point  where,  in  particular,  develop¬ 
mental  test  has  critical  shortfalls.  The  pressure  may  be 
particularly  intense  in  the  Service  budgets.  So  it  might 
not  be  unreasonable  to  expect  budget  cuts  to  be 
directed  at  T&E.  On  the  other  hand,  without  T&E 
information  how  can  leadership  make  the  decisions 
they  need  to  make?  Independent,  experienced,  objec¬ 
tive  reporting  of  real  program  progress  and  potential 
should  be  among  the  last  things  to  cut. 

The  protection  of  T&E  may  well  rest  with  the 
Congress,  where  there  has  been  continuing  support. 
Congress  has  always  been  protective  of  the  information 
that  testing  provides  and  therefore  protective  of  the 
testing  organizations  when  they  know  about  the 
problem. 


4.  Secretary  Gates  has  worried  about  a 
risk  adverse  culture  and  adversarial 
relationships  within  the  DoD.  Is  that  a 
problem  in  T&E? 

Everyone  agreed  that  whenever  you  have  individuals 
working  together  there  will  be  friction,  possibly  no 
more  in  DoD  than  in  other  organizations.  Building 
strategic  relationships  is  essential  to  successful  opera¬ 
tions.  It  is  important  to  maintain  good  relationships  at 
the  top  levels — with,  for  example,  the  Under  Secretary 
of  Defense  for  Acquisition,  Technology,  and  Logis¬ 
tics — and  simultaneously  preclude  adverse  action  from 
being  taken  against  DOT&E  action  officers. 
DOT&E  action  officers  can  tell  amazing  stories  about 
how  difficult  the  job  can  be. 

Most  panel  members  believed  that  DOT&E,  and 
T&E  in  general,  have  a  public  relations  problem.  This 
can  arise  merely  because  many  people  say  there  is  a 
problem!  To  counter  the  public  relations  problem 
DOT&E  needs  to  be  aware  of  what  flexibility  there  is 
in  the  system,  articulate  that  flexibility,  show  some 
flexibility,  and  thus  work  to  change  conventional 
wisdom. 

On  the  issue  of  risk,  it  was  pointed  out  that  Defense 
is  not  like  the  private  sector  because  there  is  not  an 
alternative  if  the  project  fails.  It  is  not  the  entrepreneur 
who  is  at  risk  if  the  system  fails,  it  is  the  combatant. 

5.  How  should  operational  T&E  adjust  to 
account  for  threat  changes? 

Opinion  across  the  panel  was  unanimous  that 
DOT&E’s  responsibility  is  to  assure  systems  are  tested 
against  the  threat  when  the  system  is  fielded  regardless 
of  whether  that  is  the  threat  in  the  requirements. 
Further,  the  Beyond-Low-Rate-Initial-Production  re¬ 
port  must  indicate  whether  the  system  is  operationally 
effective  and  suitable  in  the  environment  in  which  it 
will  be  used.  Some  systems  take  20  years  to  develop, 
and  for  those  systems  it  is  not  unusual  for  the  threat  to 
change.  But  it  is  unusual  for  the  threat  to  change  in  the 
year  before  the  initial  operational  T&E,  and  that  is  a 
reason  to  be  involved  in  the  developmental  testing.  It  is 
possible  to  respond  to  threat  changes  during  develop¬ 
ment. 

6.  What  does  it  mean  to  be  involved  early 
enough  to  enable  integrated  T&E? 

Integrated  can  have  two  meanings,  and  there  was  a 
discussion  of  both.  On  one  hand,  it  can  mean  joint. 
The  testing  must  be  joint,  or  integrated,  in  order  to  test 
acquisition  systems  as  they  will  be  used  in  the  state-to- 
state  and  highly  asymmetric  warfare  environments.  In 
missile  defense,  integration  is  so  important — it  is 
mandated  from  within  the  Missile  Defense  Agency 


30(2)  •  June  2009  213 


Seglie 


(MDA)  organization.  That  kind  of  integration  does 
not  come  naturally. 

The  second  meaning  of  integrated  is  that  develop¬ 
mental  and  operational  testers  collaborate  in  the 
planning  and  execution  of  the  test.  An  industrial 
software  best  practice  does  development  very  differently 
than  the  DoD.  They  start  testing  with  a  small  group  of 
very  sophisticated  users  and  find  many  problems  that 
they  fix.  They  then  expand  to  a  larger  group  of  less 
sophisticated  users,  working  their  way  down  to  users 
who  are  “all  thumbs”  with  computers.  Throughout  that 
process  they  collect  large  amounts  of  data  on  the  user 
response.  In  contrast  to  this  situation,  software  develop¬ 
ers  in  government  often  do  not  know  who  the  user  is, 
much  less  collect  data  on  what  the  user  is  doing  or  if  the 
user  is  happy.  When  considering  that  situation  it  is  clear 
that  the  DoD  has  a  long  way  to  go  and  DOTScE  should 
be  part  of  that  journey. 

7.  The  Defense  Science  Board 
recommended  developmental  T&E 
should  be  strengthened.  Do  you  agree? 

The  information  available  at  milestone  B  is  often 
critical.  Testing  has  to  be  built-in  from  the  very 
beginning.  It  is  Developmental  Test  (DT)  that  has 
information  about  technical  risk.  DT  has  to  be 
strengthened  to  report  the  critical  information  when 
it  is  most  meaningful.  Equally  important  is  that  DT 
have  an  independent  voice,  a  seat  at  the  table,  and  an 
audience  willing  to  listen.  DT  is  rooted  in  system 
engineering.  But  also  expressed  was  the  view  that  we 
will  be  no  better  off  unless  an  effective  linkage  between 
developmental  T&E,  systems  engineering,  and  reli¬ 
ability  is  created. 

8.  What  is  the  meaning  of  early 
operational  test  (OT)  involvement,  with 
competitive  prototypes? 

Prototyping  in  aircraft  may  be  different  from 
trucks  or  tanks  or  radios.  Operational  assessment  is 
very  important,  but  for  large  complex  systems 
prototyping  to  do  more  than  demonstrate  a  concept 
or  fundamental  technology  is  nearly  impossible.  The 
information  at  milestone  B  is  often  critical — OT 
must  be  involved  to  cure  the  problem  of  underes¬ 
timating  technology  maturity.  But  in  the  past  there 
has  been  resistance.  Recent  experience  has  been 
somewhat  different,  and  the  Operational  Test 
Agencies  are  changing  the  roles  and  engaging  early. 
One  example  of  early  involvement  was  the  program 
during  Secretary  Perry’s  tenure  of  doing  Advanced 
Concept  and  Technology  Demonstrator  assess¬ 
ments. 


9.  Should  there  be  an  increase  in  the 
T&E  workforce  to  strengthen  DT? 

The  panel  talked  about  many  aspects  of  the  problem 
and  solution.  They  unanimously  thought  it  was 
appalling  what  has  happened  to  the  DT  workforce  in 
the  field,  where  we  find  the  government  depending 
upon  contractor  data.  We  need  enough  knowledgeable 
people  at  least  to  know  if  you  have  a  competent 
contractor.  DoD  often  does  not  even  have  that.  The 
funding  to  reverse  this  situation  will  be  difficult. 

The  leadership  role  is  with  the  Office  of  the 
Secretary  of  Defense  (OSD)  to  create  a  viable  plan  to 
start  reconstituting  the  workforce.  The  Services  have 
plans,  but  they  will  not  get  the  people  unless  pressure 
comes  from  the  top — in  OSD.  Without  addressing 
workforce  issues,  do  we  have  the  talent  needed,  for 
example,  to  fulfill  the  important  reliability  initiative? 

10.  What  about  the  DoD  Test  Resource 
Management  Center  (TRMC)? 

Two  of  the  panelists.  Jack  Krings  and  Tom  Christie, 
had  significant  roles  in  the  establishment  of  the 
TRMC.  Both  felt  that  the  original  plan,  which 
involved  giving  budget  authority  to  the  TRMC,  was 
essential  to  enable  the  TRMC  to  carry  out  its  very 
difficult  mission.  They  recalled  that  originally  the  DT 
office  also  had  authority  over  the  Major  Range  and 
Test  Facility  Bases  and  agreed  that  it  is  difficult  even  to 
get  a  comprehensive  report  about  Service  needs.  It  is 
also  true  that  DOT&E  still  has  a  role  (mentioned 
earlier)  with  respect  to  resources  for  operational  T&E. 
One  of  the  difficulties  for  the  TRMC  is  the  layers  of 
approval  they  must  go  through.  The  organizational 
solution  should  consider  an  enterprise  approach. 

11.  Are  OTs  too  hard,  too  easy,  or 
about  right? 

There  is  no  ready-made  answer.  It  depends  on  the 
system  and  the  mission.  For  example,  the  F-35  is  a  very 
complex  system  and  will  require  many  sorties  to  have 
the  data  necessary  to  evaluate  system  effectiveness  and 
suitability,  including  the  electronic  warfare  environ¬ 
ment. 

Early  in  the  history  of  DOT&E  there  was  talk  of 
having  a  Milestone  IV  to  have  a  feedback  to  see  if 
DOT&E  got  it  right.  We  found  many  fielded  systems 
with  actual  mission  operations  that  were  different  then 
what  we  tested.  The  Bradley  Fighting  Vehicle  fording 
capability  comes  to  mind.  We  focused  on  Bradley 
effectiveness  in  fording  operations,  but  never  tested  a 
tube-launched,  optically  tracked,  wire-guided  missile 
configuration  because  the  role  of  the  system  changed 
after  fielding.  Maybe  we  need  more  time  with  the  end 
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user  before  we  commit  to  OT  plans.  And,  we  need  a 
continuous  feedback  loop. 

We  saw  many  examples  where  we  learned  much  in 
OT.  Good  OT  ought  to  be  really  easy;  if  DT  is  done 
well  there  are  no  surprises.  We  should  try  everything. 
The  F  18-E/F  OT  was  straightforward  because  of 
robust  DT. 

What  do  we  do  when  we  do  not  get  it  right,  like  not 
recognizing  the  threat  of  explosively  formed  projec¬ 
tiles?  We  have  a  critical  job  and  must  reduce  the  risk  in 
such  situations.  We  need  classic  sequential  processes 
and  concurrent  DT/OT. 

12.  Should  the  director,  OT&E  have  a 
statutory  5-year  term? 

Phil  Coyle,  who  served  8  years  as  DOTScE,  said 
that  5  years  would  have  deprived  him  of  many  years  he 
served  in  that  position,  and  the  legislation  should  say  at 
least  5  years. 

Others  reiterated  an  earlier  point  that  the  right 
relationship  with  the  senior  leadership  is  critical  and 
fixing  a  term  does  not  ensure  that  relationship. 

13.  The  last  question:  What  is  the  next 
action  DOT&E  should  initiate — the  most 
important  thing  DOT&E  should 
pursue  now? 

A  number  of  areas  were  identified  as  important  for 
the  future.  The  answers  were: 

•  Bridges  to  the  warfighter.  Working  under  Secre¬ 
tary  Gates,  there  is  an  opportunity  that  stems  from 
instances  of  lack  of  support  in  certain  military 
departments  for  the  warfighter.  DOT&E  should 
build  new  bridges  to  the  warfighter;  Secretary 
Gates  would  find  that  appealing. 

•  Joint  testing.  There  are  no  real  requirements  for 
joint  testing.  Nevertheless,  system  interdepen¬ 
dencies  are  now  more  important  than  ever.  It  is 
absolutely  necessary  to  do  interdependency  test¬ 
ing  and  it  will  not  happen  naturally,  but  must  be 
leadership-driven.  This  testing  would  be  similar 
to  interoperability  testing. 


•  A  better  model  for  the  acquisition  and  testing  of 
defense  equipment  and  services.  Systems  are 
acquired  by  some  very  new  acquisition  means — 
some  call  it  slogan-based  acquisition,  spiral  based 
acquisition,  for  example — you  need  a  compatible 
flow  of  Operational  Testing  to  match  the 
acquisition  flow.  The  high  value  of  testing  comes 
not  from  verification  but  from  discovery.  The  cost 
of  discovery  goes  up  an  order  of  magnitude  with 
each  successive  phase.  The  process  of  testing 
should  be  built  around  the  value  of  testing. 

•  Early  involvement.  The  OT  community  must 
focus  on  early  involvement,  especially  in  the 
requirements  process.  There  is  a  manpower  issue 
to  resolve:  OT  agency  staff  levels  were  not 
predicated  on  such  early  engagement. 

•  Information  technology  and  information  assur¬ 
ance.  Information  systems — and  especially  infor¬ 
mation  assurance  of  complex  software  systems — 
constitute  a  pressing  area  where  DOT&,E  must 
focus. 

The  answers  of  the  four  Directors  of  Operational 
Test  and  Evaluation  reported  above  offer  an  insight 
into  defense  leadership  processes  and  T&,E  consider¬ 
ations  that  we  rarely  have  the  opportunity  to  see  in 
such  a  comprehensive  manner.  Their  answers  offer 
important  insights  into  challenges  the  defense  T&.E 
community  has  faced,  and  they  may  portend  the  nature 
of  challenges  that  lay  ahead.  It  is  indeed  rewarding  that 
ITEA  convened  this  special  event.  □ 


Dr.  Ernest  Seglie  is  science  advisor,  Office  of  the 
Director,  Operational  Test  and  Evaluation  (DOT^E), 
the  Pentagon,  Washington,  D.C.  He  provides  scientific 
and  technical  guidance  on  the  overall  approach  to 
Department  of  Defense  (DoD)  evaluation  of  the  opera¬ 
tional  effectiveness  and  suitability  of  major  DoD  weapon 
systems,  provides  technical  review  of  test  reports,  and  serves 
as  chief  technical  advisor  to  the  Director,  DOT^E. 
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